Real-Time Feature Matching for the Accurate Recovery of Augmented-Reality Display in Laparoscopic Videos

Augmented-Reality (AR) displays increase surgeon’s visual awareness of high-risk surgical targets (e.g., the location of a tumor) by accurately overlaying pre-operative radiological 3-D model onto the intra-operative laparoscopic video. Existing AR systems are not robust to sudden camera motion or prolonged occlusions, which can cause the loss of those anchor points tracked along the video sequence, and thus the loss of the AR display. In this paper, we present a novel AR system, integrated with a novel feature-matching method, to automatically recover the lost augmentation by predicting the image locations of the AR anchor imagepoints after sudden image changes. Extensive experiments on challenging surgical video data are presented that show the accuracy, speed, and robustness of our designs.